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treated as an error pattern whose syndrome forms the compressed data. It is 
shown thaL syndrome-source- coding con achieve arbitrarily small distortion with 
the number of compressed digits per source digit arbitrarily close to the entropy 
of a binary memory less source. A "uuivernal" generalization of syndror e-source- 
coding is formulated which provides robustly-ef feetive, distortionless, coding 
of source ensembles. Two examples are given comparing the performance of 
noiseless miversal syndrome-source-coding to (1) run-length coding nnd (2) 
Lvnch-D-ivi >son-Schalkwijk-Cover universal coding for an ensemble of binary 
memory less sources. 


* This research was supported by the National Aeronautics and Space Administration 
under NASA Grant NSG >025 at the University of Notre Dane in liaison with the 
NASA Goddard Space Flight Center. A portion of this work was presented orallv 
at the IEEE International Symposium on Information Theory , Notre Dame, Indiana, 
October 27-31, 1974. 


u 


I 


1 


I. INTRODUCTION 

The conventional method [1-5] for using block error-correct! ng-codes to 
perform source encoding (or "dntn compression") in shown in Fig. 1. The source 
output is treated ns a received codeword, r, nnd a channel decoder for the 
selected code servee as the "source encoder." The channel decoder finds a 
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Fig. 1 The Conventional Method of Using Error-Correctlng-CoJes 
for Data Compression 

codeword x close (in nn appropriate sense) to £ and deliveia as its output the 
information sequence u corresponding to this codeword. The sequence u is then 
the "compressed data." Thus, there arc R compressed digits per source letter 
where R ic the code rate i.c., the ratio of tl>c number of Information digits 
to the total number of digits in n codcv/ord. At the user end, a channel en- 
coder is used to convert £ to the codeword x, and thus serves as the "source 
decoder. " 

Source encoding of the above type using linear error-correcting codes has 
been shown to be efficient for encoding memory less, symmetric sources under 
the Hamming distortion measure in which the average distortion is the average 
fraction of source digits erroneously reconstructed [5], For many "real" sources 
such as the output of measuring devices in space experiments, however, the 
information source is highly asymmetric with strong memory constraints. In 
such cases, the scheme of Fig. 1 generally becomes inefficient since the code- 
words of simply-implemented linear error-corrccting-codcs do not well-approximate 
the likely set of source output sequences. Moreover, in any case, the con- 
ventional scheme of Fig. 1 has the disadvantage for many applications that the 
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more complex device, vis., the channel decoder, Is located at the source side 
where one wishes to use the least equipment, while the simpler device, vis., 
the channel encoder, is located at the ur.cr side. 

In thin paper we describe an ItcrnutJve method for using linear error- 
corrccting-codcs to achieve data •. oresnion. This scheme, which we shall 
refer to as syndr ome-so urce -coding , is diag rammed in FJg. 2* The fundamental 



Fig. 2 The Syndrome Source Coding Method of Using 
Error-Correcting-Coues for Data Compression 

principle of syndromc-source-coding is that the source output Is treated as a 
channel error pattern, c, rather than as the received channel codeword. A 
syndr""»'“- f o'TK>r , "bleb if c simple linear device of the came complexity a." ar. 
encoder for the code, seives as Die "source encoder." The syndrome ij, i.e., 
the pattern of parity check failures, is then taken aB the compressed data. 
There are 1-R compressed digits per source letter since the number of syndrome 
digits equals the number of redundant digits in a codeword. At the user end, 
an "error-pattern-estimator" for the code, i.c., n device which produces a 
likely (in an appropriate sense) error pattern £ consistent with £, is used ns 
the "source decoder." Tills device is the heart of the syndrome decoder for the 
code, and dominates the complexity of the decoder in most cases. 

Since the error patterns which are correctable by the decoding schemes 
traditionally considered in channel coding studies tend to be asymmetric (e.g., 
for binary codes, the set oi correctable error patterns usually includes all 
sequences with a sufficiently small number of l's) and lend to match memory 
effecti (e.g., the correctable errors often include all "bursts" in which all 
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thu I'm in the error pattern arc confined to Home small span of conscculivc 
positions), the uyndromu-cource-coding method appeara well-suited to the 
compression of many real sources. In oilier words, the set of error-patterns 
correctable by simply -implemented decoders for known block codes seem to 
approximate well the "likely" set of source output sequences lor many real 
sources. Moreover, syndrome-sour cc-coding has the advantage for many applications 
that the simpler device, viz., the uyiul rone-former , is located at the source 
side while the more complex device, viz., the t v - vor-pntfern-estit ator , Is 
located at the user side. 

For simplicity, we restrict ourselves hereafter l o the binary case so 
that the source, output digits and the code digits ure in the finite field GF(?). 

In Section 11 . e give a simple analysis of syndrome-source-coding and show that, 
for a memory less binary source and for arbitrarily small distortion, the required 
number of transmitted digits per source letter can be made arbitrarily close 
to the source entropy. In Section 111, we describe a "universal" form of 
jyndrome-source-coding v.’hich is the major practical contribution of this note. 

In Section IV, we compare the performance of nniversal-syndromc-source-coding 
for memorylcss binary sources to the performance of a well-known universal 
coding scheme and to run-length coding. Finally, in Section V, we trace the 
development of syndroine-source-coding and show its relation to certain other 


source-coding methods. 
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II. SOME THEORETICAL CONSIDCR/TIONS 

With any binary sovrce, we associate the addit ive c hannel in which the 

source output forms the error pattern, i.c., in which the received word is 

_r m x + e where x is the transmitted word, where £ io the source output which 

is assumed to be statistical ly independent of x, and where the addition is 

component-by-component in GF(2). A given syndrome decoder for u given linear 

code to be used on this channel would comprise two devices, viz., the synd rome- 

T 

former that computes s ■ r H where H is the parity-citeck matrix of the code so 
T T T 

that £ * £ H ■ (>c + c)H ■ £ li , and the error - pat tc rn-e st imat or whose input 
is £ and whose output is the estimate £ of £. The corresponding estimate £ of 
x is, of course, x ■ £ ~ £ . Suppose that P e is the average fraction of digit? 
in x (or, equivalently, in e) which arc incorrectly decoded. We have then as 
an immediate consequence of the syndromo-source-coding configuration shown in 
Figi --c 2: 

Theore m: The average Hamming distortion fov syndi ome-source-coding of a 

given binary source coincides with the per-digit error probability P^ when the 
corresponding syndrome decoder is used with the given linear code on the additive 
channel associated with the source. 

We note that the above theorem applies to all lr' r codes, whether ilock 
or convolutional, since rothlug prevents x and e from being semi-infinite 
vectors. 

To illustrate the use of this theorem, consider the binary memoryless source 
for which the probability of emitting a 1 is p. The entropy of this source is 
H ■ - p log^ p - (1 - p) log^ (1 - p) . The associated additive channel is Just 
the binary symmetric channel with crossover probability p. The capacity of 
this channel is C B 1 - H. It is well-known that, for any c > 0 and any 6 > 0, 
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there exists a block length n mid a linear block code of rate R > C - £ such 

A 

that P c < c for the Ryndrome decoder in which £ is the minimum weight solution 
of ib ■ £ H (maximum weight if p > 1/2), see, e.g., [6, p.206]. But, since 
1 - R < l-C + i»lt + 6 is the number of compressed digits per rourr.c letter 
when this code and decoder arc used in syndrome-source-coding of the name source., 
we have: 

Corollary : For a memoryleaB binary source with entropy II, given any r > 0 and 

any 6 > 0, there is a syndrome-source-coding scheme based on a linear block 
code that achieves average Hamming distortion less than i and utilizes less 
than H + 6 compressed digits per source letter. 

It should be evident that the above corollary remains true if "block" is 
changed to "convolutional." The above corollary furnished some corroboration 
of our claim that the net of correctable error uatterns for linear codes wcll- 
approxlmates the set of typical sequences for asymmetric sources. 


III. DISTORTIONLESS UNIVERSAL SYNDROME-SOHRCE-CODING 


Cuided by the concepts of "uulvcrHal noiseless coding" [71, we now Intro- 
duce n generalization of syndrome- source-coding that permits the same Hourcc 
coding t> rheme to compress many different sources effectively. 

Lot V denote the vector apace of all 2 n binary n-tuplcs. Let Vj , V^, ... 
V M , M - 2 m , be a act of linear block codes of length n (i.e. Hubspacca of V) 
with rates R^ _> ... >_ R^. Let Ej , I^t ... be a partition of V such 

that the n-tuplcs in Ej all hove distinct syndromes relative to the code V^, 

i.e. all fall into distinct coscts of V . We shall ordinarily take V - {0) 

1 ■ m 

so that such a partition of V is certainly possible. By noiseless universal 

syndrome- source-coding (NUSSC) wc mean the coding scheme for a binary source 

in which the source output n-tuple e is encoded as an m bit prefix identifying 

T 

the index i such that c c E^, followed by the n(l - P^) bit syndronc is » e 

where is a parity-check matrix for V^. At the user Bide, the sourc^ia 

reconstructed without distortion by using the prefix to identify which code's 

error-pattern-estimator should be applied to the syndrome b_ to yield £. 

Normally, one would choose the partition E^, ... E^ in such a way thst 

the most likely source sequences lie in the leading blocks of this partition 

because the average syndrome length is then minimized. However, some compromise 

with such a strict assignment rule may be dictated by the desire to simplify 

the requisite error-pattern- estimators. 

A substantial simplification can be made at the source-encoder side when 

^ V M _j o “ (0). In this case, the syndrome for is Just « ■ c 

and hence is trivially formed. Letting k^ k^ i ... — Si 1 ^ ,c co< * e 

dimensions, we note that the nesting of the codes implies that the parity-check 

matrix 1L, , for code V., , can be selected no that its first n - k. rows form 
M-l M-l 1 
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the parity-check matrix for code V^, 1 < i < )!• Thus, only the syndrome- 
former for code j need be implemented n« its first n - k^ digits will be 
the desired syndrome ir when e c for 1 <_ i < M. 

It would now be easy to prove theorems on the effectiveness of NUSSC, say 
for the ensemble of memory less binary sources. But we believe the key concepts 
as well as the practical potential of NUSSC will be made more apparent from 
some examples in which NUSSC is compared to some well-known data-compression 
schemes. 





IV. EXAMPLES AND COMPARISON 


As a fl.st example of NUSSC, take n ■ 15 and M • 4, and let (1) " V 

be the trivial (15,15) code such that ■_ Is tSc empty string, (2) be the 
(15,11) Hamming code, (3) V be the (15,7) doublo-error-corrccting BCH code, 

v 

and (4) ■ (Oj be the trivial code such that £ ■ £.• For the partition of V 

to be used, let (1) Kj be (0), (2) E 2 be the 15 n-tuples of weight 1, (3) E. 
be the 105 n-tuplcs of weight 2, and (4) E^, be the n-tuples of weight 3 through 
15 Inclusive. Because - (0), we note that, by our previous 

discussion, only the syndrome-former for codr need be implemented. We also 
note that the error-pattern-estimators for and are trivial, that for 
is a simple threshold decoder [ 8 , p.95], and that for is a simple combinatorial 
device. 

In general, for NUSSC, the average number of compressed digits per source 
letter, v, is given by 

V - J + (1 - R x ) Pj + ... 4 (1 - R M ) P M (1) 

where is the probability that the source sequeucc £ will lie in Ej. For 
our example and for the memory less binary source considered above, P^ - (1 - p)^ 3 , 
P 2 - 15p(l - p) 14 , P 3 - 105p 2 (l - p ) 13 end *4 " 1 - p j “ p 2 " p 3 * Using these 
values in ( 1 ) and defining the efficiency, n, of the source coding scheme by 



(which is the ratio of the smallest average number of compressed digits per 
source letter that suffice for distortionless source reconstruction to the 
average number in the given distortionless coding scheme), we can calculate the 
efficiency of the NUSSC of our example. The results of this calculation arc 
shown in Figure 3 for the ensemble of memoryless binary sources with .01 <_ p <_ .5. 
We see that this particular NUSSC scheme is quite robust, maintaining an 
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efficiency of about 50Z or more over the entire source ensemble and an 
efficiency of 80Z or more over the interesting range .045 <_ p .5. 

Por comparison to NUSSC, we alao give in Figure 3 the efficiency of run- 
length (RL) coding and the efficiency of Lynch-Davisaon-Schalkwijck-Cover 
(LDSC) coding. 

In RL coding, the coder transmits an m-blt number giving the radix-2 form 
of the number of consecutive 0's (possibly none) between each 1 emitted by the 
source; except that, when this run-length is 2 m - 1 or greater, the coder then 
transmits the radix-2 form of 2° - 1 followed by the code for the remaining 
number (possibly zero) of 0's in the run. 

LDSC coding which was the earliest "universal" noiseless coding scheme rnd 
remains one of the ir >st useful, operates as follows. Hie code for n source 
digits, n ■ 2 m - 1, is an in-bit prefix giving the radix-2 form of the Hamming 
weight, w, of this n-trple followed by flog^((”) - lf| bits giving the rank of 
the particular weight w sequence in the lexicographical ranking of the (^) such 
n-tuples. (Here, [” ] denotes the smallest integer not less than the enclosed 
number.) This scheme was first described by Lynch (9] whr gave a ranking 
algorithm. Da\isson [ 10 ] noted its "universal" character and gave an inverse 
for the ranking algorithm. Later, Schalkwijk [ 10 ] simplified the ranking 
algorithm, while Cover [ 11 ] simplified the inverse algorithm and also generalized 
the algorithms for use with other sets of sequences. 

In Figure 3, we show the efficiency of RL coding with m - A (in which runs 
up to length 15 have a single A biL code) for the binary r’omoryless sources 
with .01 < p < . 5 . We show also the efficiency of n ■ 15 LDSC coding for the 
same source ensemoie. We note that n - 15 NUSSC significantly outperforms both 
other coding methods over most of the given range of p. The complexity of 
implementing the NUSSC scheme of Figure 3 would appear to be intermediate 
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Fig. 3. Efficiencies of NUSSC, I.DSC, nnd RL coding of length 15 

sequences from inemoryless binary sources with probability 
p of emitting a "1". 



Fig. 4. Efficiencies of NUSSC, I.DSC, and RL coding of length 31 

sequences from memoryless binary sources with probability 
p of emitting a "1". 
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between that of the simple RL cod'ng scheme and that of the LDSC coding scheme. 

By way of analogy to I.DSC coding, one can view NUSSC an a similar Indexing 
of n-tuplcs where the index of an n-tuple £ In the act F. Is taken an its 
syndrome s under the parity-check matrix , rather than as its t ..k In none 
ordering of the sequences in E^. The linear mapping from r to s would seem 
generally simpler to implement than a lexicographical ranking algorithm for a 
comparably-sited set of scqucrccs, although in general the former mappirg does 
not use the minimum number of encoded bits while the latter always docs. The 
greater efficiency of n ■ 15 NUSSC over n • 15 I.DSC coding as shown In Figure 3 
might thus seem paradoxical, but is explained by the fact that reduction of the 
prefix length from 4 to 2 for the NUSSC compared to LDSC, because the former 
partitions V into 4 rather than 16 subsets, more than compensates for the 
•’Mr-tly less efficient indexing of the sets. Indeed, the practical potential 
of NUSSC would Bccm to reside in the vealth of possible partitions of V whose 
component sets can be simply indexed by syndromes of linear codes. 

As another example of NUSSC, we take n - 31 and M ■ 8 and choose: Vj ■ V 
and E x - {0}; V 2 , V 3 and as the (31,26), (31,21) and (31,16) BCD codes with 
E 2 , and as the n-tuples of weights 1, 2 and 3 respectively; as the 
(31,11) BCH code and E^ ns the n-tuples of weights 4 and 5; V & as the (31,6) 

BCH code and Eg as the n-tuples of weights 6 and 7; as the (31,1) BCH code 
and Ej as the n-tuples of weights 9 through 15 inclusive; nnd Vg ■ {£} and 
Eg as the n-tuples of weight 16 nnd greater. 

In Figure 4, we show the efficiency for this n - 31 NUSSC scheme, as well 
as the efficiencies of m ■ 5 RL coding and n ■ 31 LDSC coding, for the initio 
ensemble of sources ns used in Figure 3. 0u r discussion of Figure 3 applies 
almost verbatim to Figure 4. We do note, however, that, ns can be seen by 
comparing Figures 4 and 5, the n ■ 31 NUSSC tends to give more nearly uniform 
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performance than n ■ 15 NUSSC although with a Hranllw? "peak efficiency"; In 
a sense, the n - 31 NUSCC scheme is the more "universal." 
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V. HISTORICAL BACKGROUND AND RKMARKS 

The first explicit use of "syndromc-source-coding" appears to be by 
Ohnsorge [13) who considered using the syndrome, of a length-n, t-crror- 
correctiug, cyclic code to code n-tuples of weight t or less. Fung, Tavares 
and Stein [ 1A ) used a very similar procedure, except that, rather than always 
coding source sequences of length n, the- coded n' source digits with n - n' 
dummy 0's appended when there were t l's in the first n' positions and n' <_ n. 

Tills pre-conditioning of the source results In distortionless encoding. There 
seems to have been no prior use of "universal" syndromc-source-codlng, however. 

Less explicit, but earlier, use of what can be interpreted as syndrome- 
sourcc-coding was made by Blizard [15] who conrldered "convolutional coding" 
of a binary source with code rate greater than unity (so that compression is 
achieved) coupled with a sequential decoder wnosc "metric" is determined by 
the source statistics. Forney [16] also considered, at about the same time, 
a similar scheme but discarded it as impractical because of the heavy computational 
load on the sequential oecoder; Forney did, however, interpret the encoded 
sequence as a syndrome sequence for a code of rate less than unity. More recently, 
there has been work on joint source-channel encoding which is similarly related 
to syndrome-sourcc-coding. Koshelev [17] has studied the encoding of sources 
by a convolutional encoder (without the usual pre-encoding to remove spurce 
redundancy before channel coding) in which the encoded output is directly trans- 
mitted through the channel and the source sequence is recovered by sequential 
decoding. He proved that it is possible to obtain arbitrarily small average 

Hamming distortion whenever the rate R of the code is less than R /H , 

° corap comp 

where R is the usual computational cutoff of the channel and H is a 
comp r comp 

quantity depending only on the source (H CQ > H.) Heilman [18] pursued the 
same approach as Koshelev, but noted that, in the noiseless case when 
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*comp ■ R woul<1 normally be greater than 1 and the achenc would be performing 
data compression. Using random coding arguments, Heilman showed thdt there 
exist convolutional codes for memory less sources that obtain arbitrarily small 
uvera b “ Hamming distortion with the number of compressed bits per source letter 
arbitrarily close to the source entropy. The syndrorae-source-coding interpre- 
tation, as given in Section II, provides a perhaps simpler path to the same 


result 
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